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(57) ABSTRACT 

An N session distributed architecture provides a software 
solution to the major computational challenges faced with 
providing secure communication. A registration entity is 
identified as the session arbitrator through which N devices 
on a network dynamically participate in establishing, main- 
taining and destroying cryptographic sessions. Session keys 
are generated by one or more devices registered with the 
registration server. Multiparty key agreement is used to pass 
session keys to all parties involved in the encrypted session. 
All sessions appear to be local to the arbitration server, 
however individual session are maintained by several 
devices operating as a collective. Encrypted stream parti- 
tioning and computational resource allocation to decrypt the 
individual partitions in such way as to ensure system sta- 
bility with increasing session demands is introduced in the 
architecture. This provides a cryptographic system architec- 
ture with encryption/decryption processing power limited 
only by the number of participants in the collective and 
network bandwidth or latency. 

10 Claims, 15 Drawing Sheets 



Bandwidth Slicing Block Diagram 
Ciient/Server 

Point to Multi-Point to Point Model 
8 agent model shown (can be extended to N agents) 



startIlan/mobile/extranetI WAN/internet | gateway | intranet/corporate NETi/voRK 



END 



Client 
sends/ 
receives 
encrypted/ 
ecrypted 
packets 

710 



Packets / 
-1.2.3,4r1 
5,6.7,8 



Packets 
"1.2.3,4 



IPacketSj 
5.6,7.8 




720 



Packets 1 — ^ 



1 

Packets 2 — ^ 



Packets 3* 

Packets 4 - 
Packets 5 _ 

Packets 6 - 

Packets 7- 

PacketsS- 



2 

4 
5 



740 



> Packets 
1,2 



' Packets 
3,4 



\packets 
h ,2,3.4^ 



\ Packets 



> 



H,2.3,47 
/ 5,6.7,8 



Packets 
5,6 



\ Packel 
/ 5,6.7.8 



Server 
receives 
/sends 
decr/pted 
/encrypted 
packets 

770 



— \ 

^ VPac 
7 



Packets 



760 



750 



Agent 

•Packets 9-N 
follow round 
robin algorithm 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 1 of 15 US 6,484,257 Bl 



Test 1 Configuration 



125 



Destination Client2 



BottleNeck c 



130 



NO AGENTS/2 HOSTS 



Destination Clientl 




GATEWAY SERVER 
(Decrypts Traffic then 
forwards to client) 



2 CLIENTS 



Client2 



Clientl 



Fig.1 

(Prior Art) 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 2 of 15 US 6,484,257 Bl 



CryptoScale Test 



225 



230 



Agent 



2 AGENTS/ 2CLIENTS 



Destination Client2 



Destination Clienti 




2 CLIENTS 



Fig. 2 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 3 of 15 US 6,484,257 Bl 



Agent, interface and environment 



Incoming packet (input) 



Outgoing packet 
or 

discarded packet (decision) 




Fig. 3 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 4 of 15 US 6,484,257 Bl 



402, 



Main Server 



Initialize 
405 



410 

415- 



Authenticate 
" \jControl/Setup Information 



420 



435" 
440. 

445- 
450- 



460- 
465- 

470- 

Control/Comp 
475-v 

480- 



Entity/Event Diagram 
Agent(s) 

Initialize 



Client(s) 



Initialization Acknowledgment 
, / Ready ^ 



Authenticate 
Agent Notification of new Client 



Accept or deny client based 
on available resources 



Initialization Acknowledgment 
/ Ready 



Tunnels encrypted packets 
to Main Server 

455" Y Forwards packets to Agents ^ 



Tunnels encrypted packets 
to Main Server 



FonA/ards packets to Agents 



Initialize 



J25 
430 



Control/Setup Information 
Agents Assignment 



Control Message/Begin 



Processes packets 



Processes packets 



etion Message Sent/Connection Closed 
?i?.Qlro[/CJose cljent connectjon^ 

^ Ack/ Connection Closed 



Fig. 4 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 5 of 15 US 6,484,257 Bl 




08/19/2004, EAST Version: 1.4.1 



U.S. Patent 



Nov, 19, 2002 Sheet 6 of 15 US 6,484,257 Bl 




08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 Sheet 7 of 15 US 6,484,257 Bl 



CRYPTOSCALE 



Client 
agent ip table 
630 



1.5 MB 



l/Ndienttraffcon each 
agent 



'SDES 



NAT 



Backbone Router 1GBPSDES 
agent ip table I I 
620 "-^ 



100 MB 



Round 
Robin 



IPSec 



Mode 



Gateway Server 
Master ip table 
640 



PS DES 





|o| 1 o o o o o 0 1 


o o 






Intelligent Hub 
agent ip table 
610 



Agent N 



Fig. 6 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 Sheet 8 of 15 US 6,484,257 Bl 



A 



0) 



E 
5 

CO 

O 

_o 

CO 

M g 

-o 

CO 



CO 

— Z 

^ a> 
c c 

Q- X 

I! 

^ o 

C 

CO 
CO 

oo 



. CO CD © CO 
5;r <D CO ^ -Jii jii 




cu 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 9 of 15 US 6,484,257 Bl 



STOP 




DISTRIBUTED ENCRYPTION ENVIRONMENT MAIN PROCESS 
812 



AGENT 
AUTHENTICATION 
AND REGISTRATION 
PROCESS 



834 



PREDEFINED PROCESS 
FORSTARTINGAGENT 
SERVICES SEE FIG. 9 



828 



SERVER TRIES 
TO CONTACT 
AGENT 



SERVER WAITS 

FOR 
CONNECTION 



No 



Yes 



836 



-No 



No* 



ADMINISTRATOR 
ALERTED 






855 







SERVER TRIES^ 
TO CONTACT 
OTHERAGENTS 



-Yes- 



CONNECTION 
CLOSED AND 
SERVER 
NOTIFIED 



830 







AGENT 




^No- 




METHODS 






INVOKED 






1 



820 



CLIENT 
CONNECTS 



822 



CLIENT THREAD 
CREATED ON 
SERVER 



No 



■No- 



Yes 
i 



832 





SUCCESSFUL 




CLIENT 




REDIRECTION? 



No- 



Yes 



824 



CLIENT 
AUTHENTIC- 
ATION 
PROCESS 



-No-^ 



SECURE 
COMMUNICATION 
BEGINS 




840 



T 
Yes 

JL 



834 



CONNECTION 
PASSED AND 
SERVER 
THREAD FREED 




SECURITY 
VIOLATION 
LOGGED AND 
CONNECTION 
CLOSED 

(STOP) ~> 



No 



842 



PREDEFINED PROCESS 
FOR AUTHENTICATING 
AND REGISTERING 
CLIENTS FIG. 9 



PREDEFINED 

PROCESS 
FOR INVOKING 

AGENT 
METHODS AND 

PASSING 
CONNECTIONS 
BETWEEN 
AGENTS 
SEE FIG. 10 



Yes 



838 



SECURE 
COMMUNICATION 
BEGINS 



Fig. 8 



08/19/2004, EAST Version: 1,4.1 



U.S. Patent Nov. 19, 2002 sheet 10 of 15 US 6,484,257 Bl 

AUTHENTICATIONAND REGISTRATION PREDEFINED PROCESS 



1110 



AGENT 
STARTS 



MULTIPARTY 

KEY 
AGREEMENT 



905 



No- 



Yes 



SECURE 
MESSAGING 
BEGINS 



Yes 



CLIENT/AGENT 
SUPPLIES AUTH 
CREDENTIALS 



Yes 

I 



SERVER 
REGISTERS 
CLIENT/AGENT 



—] — 

Yes 
i 

STOP 



910 
U 



No 



925 

U 



No 



SECURITY 
VIOLATION 
LOGGED AND 
CONNECTION 
CLOSED 



915 

J 



930 
U 



No 



^935 

y 



920 

A_ 

STOP 



Fig. 9 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 Sheet 11 of 15 US 6,484,257 Bl 



824 



CLIENT 
STARTS 



CLIENT PROCESS 
- No 



STOP 




No 



JVes 



CLIENT 
CONNECTS 
TO SERVER 



1015 



Yes 



CLIENT 
AUTHENTICATION 
PROCESS 




PREDEFINED PROCESS 
FOR AUTHENTICATING 
AND REGISTERING 
CLIENTS SEE FIG. 9 



LOGS 
FAILURE AND 
NOTIFIES 
SERVER 



1025 



Yes 



CLIENT PASSES k 
INFO TO 
SERVER 



1030 



No 



Yes 



LOGS 
FAILURE 
AND RETRIES 



1035 



CLIENT 
REQUESTS 
UPDATES FROM 
SERVER 



1040 



1. 



No 



LOGS 
FAILURE 
AND RETRIES 



1045 



es 
1 



CLIENT 
RECEIVES 
REDIRECTION 
REQUEST FROM 
SERVER? 
TIo 

SECURE 
COMMUNICATION 
BEGINS WITH 
SERVER 



1050 



-Yes- 



1065 





SUCCESSFULLY 


^ 


REDIRECTS TO 




AGENTS? 




,Yes 




SECURE X 




COMMUNICATION i 


( 


BEGINS WITH i 




V AGENT y 


-No — 





1055 
-No- 



1060 



LOGS 
FAILURE AND 
NOTIFIES 
SERVER 



1070 



Fig. 10 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 Sheet 12 of 15 US 6,484,257 Bl 



INVOKING AGENT METHODS CONNECTIONS PREDEFINED PROCESS 

830 

CLIENT 
STARTS 



IK 




PASS CONNECTION 
CHECK 



1110 



LOGS FAILURE 
AND RETRIES 



1125 



PREDEFINED 
PROCESS 
TO DETERMINE IF 
EXISTING CLIENTS 
SHOULD BE 
PASSED TO 
NEW AGENT 
SEE FIG. 12 



Yes 




SERVER CONNECTS 
TO THIS AGENT 



SERVER THREAD 
CREATED ON 
AGENT 

Vfe 



1145 



-No- 



No 



Yes 



1150 



i. 



SERVER BEGINS MKA* 
WITH THIS AGENT 
INSTEAD OF LOCAL 
AGENT 



No 



814 



CLIENTAUTHENTICATION 
PROCESS 



No 



SERVER REDIRECTS 
CLIENT TO AGENT 



1155 



SECURITY 
VIOLATION 
LOGGED 
SERVER 
NOTIFIED AND 
CONNECTION 
CLOSED 



1165 



1170 



STOP 



No 



CLIENT THREAD 
CREATED ONAGENT 



1160 



PREDEFINED PROCESS 
FORAUTHENTICATING 
AND REGISTERING 
CLIENTS SEE FIG. 9 



*MKA-IVIULTIPARTYKEY 
AGREEMENT 



SECURE 
COMMUNICATIONS 
BEGIN 




1175 



■Yes 



Fig. 11 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 sheet 13 of 15 US 6,484,257 Bl 



PREDEFINED PROCESS FOR PASSING CONNECTIONS 
1110 




1205 



1215 



1220 



AGENT NOTIFIES 
SERVER 


No 


LOGS FAILURE 
AND RETRIES 





SERVER AND AGENT 
EXCHANGE CONNECTION 
PASSING INFO 



Yes 
i 



SERVER SENDS CLIENT 
REDIRECTION 
INFO 




SECURITY 
VIOLATION 
LOGGED 
SERVER 
NOTIFIED AND 
CONNECTION 
CLOSED 



1230 



E 



1245 



CLIENT REDIRECTS 
SUCCESSFULLY? 



-No- 




1240 



STOP 




1250 



Fig. 12 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent Nov. 19, 2002 Sheet 14 of 15 



US 6,484,257 Bl 



1310 



1320 



Input Agent-| 




Composition Restrictions 



Agent: 



Agents internal functions 
don't rely on each other 



Agent k 



Output Agent-] 



Output Agent 
Input Agent 1^ 



Output Agent 1^ 



1330 



System Behavior Cannot Be 
Measured/Characterized 




Outputs from one agent are 
never inputs for another agent 



Output Agenl|^ 



Agentk 



Internal Trace 



Input Trace 





Output Trace 



Fig. 13 



08/19/2004, EAST Version: 1.4.1 



U.S. Patent 



Nov. 19, 2002 Sheet 15 of 15 



US 6,484,257 Bl 




08/19/2004, EAST Version: 1.4.1 



us 6,484,257 Bl 

1 2 

SYSTEM AND METHOD FOR MAINTAIN FNG noQ-linear cost difference, in installing and maintaining a 12S 

N NUMBER OF SIMULTANEOUS k Frame Relay link versus a 1.544M Frame Relay link. The 

CRYPTOGRAPHIC SESSIONS USING A cost problem is not limited to bandwidth, but rather is also 

DISTRIBUTED COMPUTING greatly affected by the addition of new groups of hosts as 

ENVIRONMENT 5 additional connection points. Related equipment also must 

be installed and maintained. Point-to-point encryption also 
has cost disadvantages. Point-to-poinl link level encryption 

FIELD OF THE INVENTION is usually all or none meaning that all data both public and 

. . , „ , private are encrypted over this link. This additional overhead 

Itie held or the present mvention relates generally to the • .t* • i - u * j • ui • *u 

. . . f 30 IS acceptable m some cases but undesirable m others, 

encryption and decryption of data conducted over a distrib- ^incc link level encryption requires static routes to be 

uted computer network. In particular, the field of the™ ^^ ^^^^ -J .^^^ ^^^^ ^^^^^^^^ 

tion relates to a software architecture for conducting a ^^^^^ ^^.^^ ^^^^ ^ dynamically routed 

plurabty of cryptographic sessions managed over a distrib- ^^ j„, ^ ^^^^^^ ^ . ^ encryption 

uted computing environment. ^^j^^^^^ ^ i^^^ i„ ^^^^^ ^^ ^^^^ j^,^ framework 

AnN session distributed architecture is described which of rout able IP packets. Currently there exists a transport 

solves the problems encountered with providing a secure level security mechanism for appUcation programs using 

network. The present software solution boosts performance SSLv3 (secure sockets layer). SSL was developed in 1995 

to previously unattainably high levels and provides a prac- when a universally recognized security mechanism at the IP 

tical security solution capable of servicing N simultaneous 20 layer did not exist. This has been the most commonly used 

cryptographic session using a distributed computing envi- protocol for providing secure applications, 

ronment without additional encryption decryption hardware The three protocol capabilities of SSL include 

at wire-speed levels. An aspect of the invention provides a authentication, encryption and key exchange. In IPSec these 

solution, which overcomes the network bandwidth latency are provided as separate protocols (AH, ESP and IKE), 

barriers to secure encryption. Another aspect of the inven- 25 In SSL most of the communications protocol data is 

tion provides a scalability solution to the problem of pro- passed in plaintext, only the application header and actual 

cesser saturation due to encryption decryption loads. data sent to the application is cryptographically protected. 

The encryption and integrity protection for the data and not 

Drt.v.,jvvji\wui\u (jjg communications as in IPSec, which protects both, are 

There is a growing need to provide for secure commerce handled by the record protocol. The negotiation of new 

on computer networks, which does not require cosUy non- ^^yP^^ algorithms and keys is handled by the handshake 

scalable computaUonal resources. Corporations now have protocol. Fmally, any errors that have occurred are handled 

critical needs for ensuring the security of data that traverses ^^^^ protocol. SSL maintains its security state based 

their networks. Information Systems (IS) managers have on the session associated with a particular set of host 

attempted to cope with those needs by installing and man- addresses and ports. 

aging expensive hardware to provide protection of data. Id sessions are established in four steps. In Step 1 the 

the case where data must be transferred between sites, IS render sends a hello message to the receiver containing 

managers can dictate their security needs to the telephone random data. In Step 2 the receiver sends the sender his/her 

companies who manage the transfer of data between mul- P^^^^^^ embedded in a signed certificate. In step 3 the 

tiple sites. However, there are several problems limiting the sender encrypts a shared secret key and a change cipher spec 

transfer of data networking. Such concerns are as follows: switch (to detennine the proper cipher to use) with the 

VT * 1 A -1 / 1 1 \ receiver's public key and sends it to the receiver. In step 4 

Network Availabihty (also known as uptime): . , , • . j . , / T 

^ 1- jjie receiver sends a reply using the shared secret key (after 
Network bandwidth (the amount of data that the overall decrypting the info in step 3 with his private key) and a 
network can handle over a particular time slice); 45 "finished" message. Both sides now can begin communica- 
Quality of Service: ensuring that pre -determined service tions. Using the record protocol, all data that passes between 
levels, such as bandwidth congestion allowances and the two parties are encrypted and hashed and the recipient 
network latency, are consistently met for all hosts checks this hash upon decryption to make sure that the data 
connected to the network; have not been modified in transit. 
Security: ensuring that sensitive data are protected as it 50 The newest version of SSL (3.0) supports RSA key 
traverses the network and those unauthorized parties do exchange, DifiSe-Hcllman anonymous or signed (the most 
not compromise that data or the network itself. common implementation is SKIP) and Fortezza using SKIP- 
Moniloring/Auditing (the capability to verify that the JACK. TLS (Transport Level Security) and PCT (Private 
above needs are being met and the ability to instantly detect Communication Technology) by Microsoft are both varia- 
and react to any deviation from preset expectations). 55 tions on SSL that are vying for standards approval by the 
When considering a new technology that will impact a IETF. A major disadvantage of all versions of SSL is that 
network, an IS manager must address the foregoing issues. SSL is ineffective against many of the newer communica- 
After these requirements are met, factors of cost and seal- tions level (below transport level) attacks, which are tech- 
ability must be considered. IS managers are constantly nically called SYN Hooding, Buffer Overruns and Traffic 
looking for ways to meet the above requirements while 60 Analysis, 
reducing the cost of supporting their network. Managing the IPSec 

cost of expanding a network to address increased bandwidth IPSec is a conventional protocol for securing IP traffic as 

requirements of users is a major problem for IS managers it traverses the Internet, an Extranet or any IP based local, 

today. metropolitan or wide area network. IPSec can be incorpo- 

Point- to -Point Encryption 65 rated with Ipv4 to provide security for host to host, host to 

Point-to-point link level encryption has a disadvantage in subnet and subnet to subnet communications, which are not 

that it is not scaleable. For example, there is a dramatic and available with SSL. 
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Tne objeciive lor securing large coiporatc iiciwuiks is iu 
allow the proper insiders or outsiders to access corporate 
data transparently while keeping unintended parties from 
accessing the same data or denying service to those who 
should be accessing the data. In the past, Firewalls have been 5 
used as a means for filtering incoming and outgoing traffic. 
Firewalls have been combined with access servers to authen- 
ticate parties before they arc allowed access to any resource 
inside or outside the firewall. 

Firewalls have evolved to include new protocols that lO 
allow them to safely transfer data between themselves and 
another party over the Internet. This function is known as 
creating a virtual private network (a private network over the 
public Internet). 

The IPSec protocol uses two underlying protocols to send 15 
data securely. IPSec adds two additional packet headers to a 
packet to handle each of the two protocols. The headers both 
contain a numerical value known as the SPI (security 
parameters index) to identify the crypto keys and procedures 
to use with it. The first header, AH (authentication header), 20 
provides integrity checking and keying information to keep 
attackers from computing alternate checksums that check 
correctly. The second header, ESP, encrypts the contents of 
the remainder of the packet. 

IPSec supports a number of algorithms for authentication 25 
and encryption. Examples are KeyedMDS and SHA-1 (for 
AH), DES, Triple DES, and RC4 (for ESP). In addition to 
this, IPSec automatically handles the creation of security 
associations between hosts through key management. 

Manual keys can be used which allow hosts to be con- 30 
figured manually with the proper shared secret keys. More 
common is the use of Simple Key Interchange Protocol 
(SKIP) which negotiates and exchanges session keys 
between IPSec hosts. ISKAMP (Internet Security Associa- 
tion and Key Management Protocol) is a general purpose 35 
protocol intended to manage security associations and man- 
age key exchanges using Oakley or IKE. Tunneling is also 
used. In tunnel mode the final destination IP header is 
encrypted and a gateway IP header is added to allow router 
to route the packet to the gateway server. In transport mode 40 
the IP header is not encrypted. 

IPSec is meant to protect traffic between hosts. However, 
with the wide range of applications currently in use (email, 
browsers, file u-ansfer, remote terminal access, multimedia, 
database and so on) it becomes cumbersome to implement. 45 

IPSec provides an advantage over SSL because it can 
protect against the newer protocol attacks such as SYN 
flooding and buffer overmns. In the SYN flooding attack 
mentioned above IPSec would block iUcgitimate SYN mes- 
sages because they require a valid AH with a valid crypto- 50 
graphic checksum. Attackers cannot generate numerous 
requests from random hosts because they cannot generate a 
valid AH for every such host. In the buffer overrun attack, 
the destination host will discard any IP packets which are not 
properly formatted for IPSec. That is, packets must come 55 
from a vaHd host and be properly formatted for TCP before 
TCP processes them, thereby protecting the host from this 
type of attack. 

Although SSL and IPSec can be combined to gain added 
protection and flexibility, these systems fail to address the 60 
problems of increased cost of implementation and scalabil- 
ity. Another major problem not addressed by SSL and IPSec 
is managing the cost of expanding a network to address 
increased bandwidth requirements of users. 

The introduction of constantly changing standards and 65 
encryption/decryption schemes has greatly increased the 
burden on computer resources to provide secure comrauni- 



addressed with the addition of special encryption/decryption 
ASICs (Application Specific Integrated Circuits) or hard- 
ware. However, as hosts are called, a doubling of hardware 
must be added to meet demands from both source and sink 
ends. 

A conventional attempt to address the above deficiencies 
includes the use of hardware to handle encryption and 
decryption of data traffic. However, this is expensive and 
slow in that it increases the computational burden on the 
CPU when encrypting and decrypting data. The hardware 
approach also has a disadvantage in that it is not scaleable. 

Alternative software architectures have been tried and 
discarded. The conventional software point to point client 
server model cannot scale adequately. Increased demand for 
secure sessions can lead to system failure as processing 
resources become unavailable on either side. Employing a 
central server model has been tried and found inadequate. 
The central gateway server in a distributed system environ- 
ment becomes saturated with increased demand for decryp- 
tion services. Tests were done to compare the performance 
of a conventional centralized server architecture model, as in 
FIG. 1, against an invention architecture topology in FIG. 2. 
Tbese configuration performance tests and results are dis- 
cussed infra 

Conventional distributed architecture is unable to manage 
the increase in secure session demand due to instability 
arising fi^om uneven processor computational loads, propa- 
gation delays and computer or network latency, all of which 
cause loss in synchronicity with collective processors. For 
these reasons current solutions are inadequate to overcome 
the barriers mentioned above. 

Therefore, what is needed is a new method for encryption/ 
decryption which is infinitely scaleable in the number of 
simultaneous sessions capable of being processed by a 
server. 

What is also needed is an encryption/decryption system 
which is infinitely scaleable in terms of bandwidth between 
clients and servers. 

What is also needed is an easily implemented software 
solution which provides end-to-end encryption/decryption 
in a distributed network while increasing processing power 
which eliminating latency as bandwidth increases. 

SUMMARY 

In accordance with the foregoing and other objectives, an 
aspect of the invention provides a distributed software 
solution for encryption/decryption which is infinitely scale - 
able in the number of simultaneous sessions capable of 
being processed by a server and in terms of bandwidth 
between chents and servers. Another aspect of the invention 
provides end-to-end encryption in a distributed network and 
combines the processing power of all computers connected 
to the system to enable bandwidth to be infinitely scaleable 
and to reduce latency substantially to zero. 

Another aspect of the invention provides a software 
architecture for encryption/decryption by partitioning the 
client traffic into units which can be processed across a 
distributed network of hosts without introducing network 
instabilities. A further aspect of the invention increases 
packets per second throughput and overcomes latency. 
Another aspect of the invention implements a mathematical 
method ensuring a stable partitioning and processing of 
encrypted traffic to meet the increase in secure session 
demand. 

In accordance with another aspect of the invention, the 
software architecture has three primary components: 
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Manager, Client and Agent. The manager software resides DET.4JLED DESCRIPTION 
on a gateway server and manages aU a^ects of controUing accordance with an aspect of invention, the manager or 
the system. Ghent, server, and agents are created on the g^rver, agent and cHent are aU designed to operate 
manager. The manager controls cUent access levels. Certifi- transparenUy within any distributed network which uses an 
cate mformation is imported and stored by the manager or s internet protocol aP). Examples of such distributed net- 
optionally generated by the manager. The manager does ^^.^s may employ Ethernet, Token Ring, Synchronous 
performance monitormg. The manger performs auditing. Optical Network (SONET), ATM, Gigabit Ethernet 
Network address translation is handled by the manager for networks, or the like. They will not disrupt network traffic 
tunneled traffic from the chent ^^^^ ^^^^ machines or on the wire. The invention 
The client software resides on the desktop of mternal ^^^^^ ^^^^ ^^^^^ technology to establish end to 
hosts, the desktop/laptop of remote users and the desktops of or "final mile^ security links to the final destination 
remote offices. The Chent software provides a simple GUI ^^^^^ business network, 
interface for clients to configure dial-up information and use Manager Initialization 

either a dial-up connection or a network connection to the Referring to FIG. 4, the manager will load the poUcy file 

local VPN server. j5 ^^^^ -j ^^^^ database) upon initialization. The IKE 

The agent software handles the negotiation of security engine will start on the server and receive setting informa- 

keys, security associations and establishes the IPSec link xion from the server daemon based on what it has loaded 

between itself and the server. Agent software can run as a from the policy file. Encryption/decryption settings will be 

stand alone process or exist as part of the client software. get, integrity checking settings will be set, rc-kcying settings 

The agent software is responsible for encrypting and will be set and access time information wiU be set on the 

decrypting communication traffic as it arrives from the manager. At this point, the server (and IKE engine) will 

clients via the server. All of the agents operate as distributed establish a connection with the agents listed in the policy file 

system to share the load of the encryption and decryption and (after establishing a secure session) upload VPN rules 

over all of the agent CPUs, (security associations, network address translation tables, 

BRIEF DESCRIPTION OF THE DRAWINGS 25 etc) to the agent. 

These and other aspects and advantages of the invention Initialization 

u ■ * *u f 11 - J * -1 J J ■ Referring to the event diagram in FIG. 4, upon mitiahza - 

may be appreciated from the following detailed descnption _c *u *• * j tmt i u 

. *u J ■ • u- u tion the agent performs an authenticated DH key exchange 

together with the drawmes in which: j r ; ui* i_ • i -^u *u ^; ^ 

^ , ^ • o ^ m order to establish a session key with the Manager. To 

HG. 1 shows a test configuration for performance mea- 3^ ^^^^^^ ^^^^^^ ^ ^^.^ ^^^^.^ 

surements for a conventional network architecture; certificates are attached to the messages exchanged (along 

FIG. 2 shows a QyptoScale test configuration for perfor- ^tj^ signatures and message digests to verify the certificates 

mance measurements in accordance with an aspect of the ^nd to make sure the message is not modified in transit) for 

invention; session key negotiation. After a session key is established 

FIG. 3 shows an agent interface and environment in 35 the manager sends the SA and vpn policy information to the 

accordance with an aspect of the invention; agent and the IKE engines on the manager and agent 

FIG. 4 shows an abbreviated entity/event diagram in exchanging keying information, 

accordance with an aspect of the invention; Client Initialization/Authentication 

FIG. 5 A shows a tunneling packet Network Layer Model Upon initialization the client contacts the gateway server 
decomposition in accordance with an aspect of the inven- ^^o and authenticates using RADIUS, TACACS+, a pre -shared 

tion; password or X.509 certificate. Once the client is 

FIG. 5B shows a final destination packet Network Layer authenticated, it negotiates the session key with the gateway 
Model decomposition in accordance with an aspect of the server. After the session key is established, the client down- 
invention; loads the VPN policy information from the server (security 

FIG. 6 shows a distributed network topology for the ^5 associations, network address translation tables, etc). With 

invention architecture in accordance with an aspect of the the VPN policy information established, the client's IPSec 

invention; engine begins communication with the gateway server and 

HG. T shows discrete packet transport across a network ultimately with the final destination, 

in accordance with an aspect of the invention; CryptoScale 

FIG. 8 shows a flowchart of the invention main process in Cryptoscale is the invenUon architecture comprised of 

accordance with an aspect of the invention; ""'""Ser or mam server, agents and clients to boost perfor- 

™^nL a u^r.L .l.-.- j mauce lo exceptional levels without the use of additional 

FIG. 9 shows a flowchart of the authentication and . , t-u- . u 1 n r. t. j ^tt^vt 

. . , ... . r .1. hardware. This technology allows software-based VPN 

registration process in accordance with an aspect of the , ^, , - . • ji 1 t-l l . . 

invention* solutions to perform at wire-speed levels. The architecture is 
A , n t r 1 f ,55 based on an asynchronous distributed model but provides 

HG. 10 shows a flowchart of the chent process m critical key synchronization within some components of the 

accordance with an aspect of the invention; architecture. The system consists of separate components, 

FIG. U shows a flowchart of invoking agent methods agents that exist as atomic objects with zero wait states, that 

connections process in accordance with an aspect of the process data in an arbitrary order and at arbitrary relative 

mvention; specks. Specific timing considerations are ignored with the 

FIG. 12 shows a flowchart for passing session connec- exception of re-keying time constraints and IP time-to-live, 

dons in accordance with an aspect of the invention; The entire system is modeled as a finite state machine. 

FIG. 13 shows the automata composition restrictions in Transitions in state are caused by an encryption/decryption 

accordance with an aspect of the invention; computation (a DES CBC block for example) on an agent. 

FIG. 14 shows the overall relationship between automata 65 There is a synchronizer component in the gateway server, 

and the automaton in accordance with an aspect of the which handles the problems that exist in managing an 

invention. asynchronous system. FIG. 4 shows the protocol and process 
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sequence, which transpires between the manager, agent(s) 
and clienl(s) cuiiiies. in the timing model, the exact order 
can be altered without departing from the scope of the 
invention. For instance, the FIG. 4 shows that the agent 
initializes and authenticates before the client initializes and s 
authenticates. This order is not limited to the precise 
sequence shown in FIG. 4 and some steps can be inter- 
changed without afifccting the encryption/decryption aspect 
of the invention. 

Timing Model lo 

Below is a discussion regarding the events that transpire 
between the main server, agent(s) and client(s). Only an 
abbreviated number of the events mentioned are shown in 
FIG. 4 and only to impart the very basic ideas in establishing 
secure session environment, initiating secure sessions, trans- 15 
f erring secure sessions and terminating secure sessions. 

First the Main Server starts up, wherein a registry is 
created and initialized and the server begins execution 402. 
The Agent Serve r(s) register themselves 405 with the Main 
Server and defiiie session key(s) with which to establish 20 
secure communications. The Main Server and Agent Servers 
become enabled to receive secure connections from Clients 
410 and 415. The Client(s) connects to the Main Server and 
authenticates using one of several servers known authenti- 
cation methods 420. The Main Server determines if it can 25 
accept a new session based on its current available processor 
bandwidth. If the Main Server can accept a new session 
based on available processor resources, then it agrees on a 
secret session key with the Client(s) and begins the session 
(s). If the Main Server has insufficient resources to service 30 
the session 425, then it will instruct an Agent Serve r(s) to 
become unblocked [wake up] and participate in a multiparty 
key exchange between a Client, Main Server and Agent 
Server. If the Agent Server has insufficient resources it will 
notify the server that it cannot accept a new client session or 35 
maintain an existing one. If none of the Agent Servers can 
accept a new client connection then the server can handle the 
additional load or deny the connection based on configura- 
tion settings. If the Agent Server loses resources it will 
request that the Main Server pass the client connection to a 40 
new Agent which the Main Server will attempt to do. If the 
Main Server cannot pass the connection it will either attempt 
to handle the load itself or notify the client and close the 
connection. If the Agent Server abruptly shuts down the 
Main Server will automatically pass the client connection to 45 
a new Agent Server or attempt to handle the load itself and 
435 and 440. The Main Server will notify both CUent and 
Agent Server of the correct cipher to use for the session. The 
Main Server will notify both Client and Agent Server of any 
special information such as special ciphers for the different 50 
types of communication formats. The Main Server will 
notify the Client and Agent Server of each other's addresses. 
The Client and Agent will independently generate a session 
key to exchange data. The Client will then begin encrypting 
its session communication to the Agent Server (via the Main 55 
Server gateway) using the key and information obtained 
from the Main Server. The Agent Server will decrypt the 
session communication and redirect this decrypted commu- 
nication to the intended final destination. If the Client 
specifics an end peer to connect to directly, then the Main 60 
Server will attempt to redirect the Client to the peer Agent 
Server after authenticating both parties 450 and 460. The 
Main Server maintains a list of connections [sessions] and 
associated session information (session keys, etc)in the 
registry wherein a session redirect or special circumstances 65 
may require this information to initiate further action. Upon 
successfully transferring a session to an Agent Server, the 



Main Server will terminate the thread of an encr^'pied 
session communication with the Client but maintain infor- 
mation in the step previous to this one. At this point the Main 
Server will serve only to synchronize the Client and Agent 
communication via network address translation 455 and 
465. If any Agent Server currently servicing a Clients 
becomes saturated [overloaded or processor resource 
insufScient], it notifies the Main Server to pass the session 
on to another Agent Server. If the Main Server receives 
notification from an Agent Server that it is saturated, then the 
Main Server finds an alternate available Agent Server and 
passes the secret key and security association information 
corresponding to that session to the said alternate Agent 
Server and securely notifies the impacted Client to redirect 
itself to the said alternate Agent Server. The Client connects 
to the alternate Agent Server and a secure session is con- 
tinued. Upon receiving notification of a successfully redi- 
rected session from the said alternate Agent Server, the Main 
Server will notify the said saturated Agent Server that the 
said session was successfully redirected. The saturated 
Agent Server wishing to terminate the session then termi- 
nates the session after it receives confirmation from the 
Main Server that the Client was successfully redirected. If a 
client or Server signals a session is concluded then a 
termination message is sent to the Main Server which then 
registers the session in the registry as terminated 475 and 
480. 

An overall system algorithm proceeds as follows: 

1 . The client authenticates to the main server 

2. The server gets the client information including the 
bandwidth requirements to determine how many agents to 
assign to the client 

3. The server sends the client a NAT (Network Address 
Translation) table, which is used to forward IP packets 
from host to another host. The NAT table contains the 
actual IP addresses of assigned agents 

4. The server sends the agents a NAT table update to add the 
client to their existing client list 

4a. The server initiates a dataflow test to ensure that the 
microflows from each agent will be reassembled properly 
by the TCP/IP module on the destination. If packet 
re-sequencing is beyond the allowed window limits by 
TCP/IP then the server will redirect the micro flows 
through other agent or use fewer agents in the session. 

5. The client and agent perform an authenticated Dififie 
HeUman exchange in order to negotiate the session key 
and proper security association 

6. The client begins the encryption process 
Packet 1 is tunneled to agent 1 

Packet 2 is tunneled to agent 2 
Packet 3 . . . 

Packet N is tunneled to agent 1 by some mutually estab- 
lished order e.g., round robin. 
Individual packet network layer model decomposition is 
shown in FIG. 5Afor two clients outgoing secure packets. 
The packets are shown broken down by network layer model 
components and show an aspect of packet processing at the 
various locations. The tunneling packets traveling from the 
client 5A10 each contain DATA 5A20, TCP 5A22, AGENT 
IP HEADER 5A24, ESP 5A26, AH 5A28, and GATEWAY 
HEADER+AGENT HEADER 5A30 components. These 
two packets arc sent across the Internet 5A80 to the Gateway 
5A40. The Gateway then strips the GATEWAY HEADER+ 
AGENT HEADER 5A30 and preappends an AGENT ID IP 
HEADER 5A60 for a packet composed of output packet 
5A50. The encrypted stream (forward direction) is more 
generaUy, partitioned among N agents. A basic distributed 
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network topology is shown in FIG. 6 Cryptoscale Configu- 
ration. FIG. 6 depicts a simple distributed topology of the 
invention architecture. 610, 620, and 630 show potential 
sources for clients needing secure communication service. 
These all contain an agent IP table or list of available agent's 
IP addresses assigned to that particular client. Each client 
table may have a different list of agents based on client needs 
and server resource scheduling. These can then employ 
IPsec tunnel mode through Gateway server at 640, which 
contains the NAT, Network Address Translation, table and 
master IP tables and then on to assigned agents at 650, where 
decryption is accomplished in parallel fashion. 
7. Agent N decrypts the packet that was sent to it and either 
sends it up the stack (reads) clear text data or forwards 
clear text data to final destination. The decryption process 
itself is closely tied to the cipher being used. This inven- 
tion employs the established standard ciphers e.g., DES 
and RC4, and decryption of each packet would depend on 
the type of encryption standard employed. The packet 
processing is shown in FIG. SB. Again, the agent pro- 
cessed packets are broken down into boxes to show the 
individual network model layer in each packet. Note that 
the AGENT IP HEADER, ESP, AH and AGENT ID IP 
HEADER layers have been stripped off by the agent. The 
remaining DATA 5B10, TCP 5B20, are pre-appended a 
DESTINAnON IP HEADER 5B30, and then forwarded 
to the final destination host 5B40, in RG. 5B for reoon- 
sUtution of individual packets. 

N Agents working in parallel on an encrypted stream are 
shown in FIG. 7 in a more general form. Tliis figure shows 
the mode where the application data is broken into M 
discrete packets and sent through encryption XOR chain. 
These packets are encrypted separately by the low level 
process 710, The figure shows 8 such packets for an 
example. The packets traverse routers and spread as they 
follow the shortest path to the agents 720. More diffusion 
occurs as packets continue on their journey 730 Packets 
from each application part reach their assigned agent and are 
decrypted individually 740 shows the packets destined for 
the same device begin to draw closer and packet concen- 
tration occurs 760 as packets get closer to the destination. 
Decrypted packets reach the destination and are reassembled 
770. 

These general stages above can be further broken down 
into more detail. FIG. 8 through FIG, 12 contain flow charts, 
which depict the process aspect of the invention in more 
depth. The flow begins in the upper left hand comer 810 of 
FIG. 8 and continues to 1250 in FIG. 12. The process is 
discussed directly below. 

Distributed Encryption Environment Main Process 

If the main server 810 starts successfully, it proceeds to 
authenticate an agent and begin a registration 814. This part 
of the process is continued in FIG. 9. Upon finishing 814 and 
returning, the main sever checks CPU load to decide 816 
whether or not to offload work to an agent. Affirmative leads 
to 828, where the server tries to contact an agent. If no 
contact is established, then the Administrator is alerted 850 
and the server tries to contact other agents 855. That is 
accomplished at 830 by invoking the agent methods sub- 
process, which is continued in FIG. U. If the agent method 
fails to invoke, then the connection is closed 836, the server 
is notified and the server tries to contact another agent 828. 
If the agent method is successful, a client redirection 832 is 
attempted and success will lead to a client connection being 
passed from the server to the agent and a freeing of that 
server thread 834. A negative will initiate closing of con- 
nection and server notification 836. Once a connection is 
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^ass^'d and server thread freed ^34, then z secure commu- 
nication begins 838. In the event of an error, a security 
violation is logged 840 and the connection is closed. If the 
server decides not to offload the demand for service to an 
agent 816, then the server wails for a client connection 820. 
If the client connects, then 822 a client thread is created on 
the server. The server checks to see if it has sufficient CPU 
resources 816 and begins 824 a client authentication 
process, which continues in FIG. 10. If the client authenti- 
cation process is successful, then 82 secure communication 
begins. 

Authentication and Registration Sub -process 

The sub -process begins by attempting a 905 multiparty 
key agreement, MKA. A failure will prompt a security 
violation log and connection termination 915. A success will 
force the process to proceed to 910 initiation of a secure 
message communication. This leads to 925 client/agent 
supplying authentication credentials. Any failure in the 
previous two steps will be logged and a security violation 
flagged, followed by connection termination 915. Asuccess- 
ful client/agent authentication 925 leads to a registration 
event on the main server 930. 
Client Sub-process 

The client sub-process begins by attempting to connect to 
the main server 1015. If this is successftil then 814 client 
authentication sub -process is initiated as shown in FIG. 9. A 
failure to authenticate will prompt a failure log and 1025 
server notification. A successful client 814 authentication 
will lead to 1030 information passing from client to server. 
If information passing was xmsuccessful, then a failure is 
logged 1035 and the attempt retried. If information passing 
1030 was successful, then the client requests an update from 
server 1040. An update failure is logged and a retry attempt 
is made 1045. If successful, the client receives a redirection 
request from the server 1050. If redirection Ls accepted 1055, 
then the chent attempts to redirect to agent. If this is 
successful, then 1070 secure communication begins with the 
agent. In the event that redirection 1050 request failed, then 
secure communication begins with the server. 
Invoking Agent Methods Connection Sub-process 

Successfully invoking the agent authentication process 
starts a pass connection check 1110, refer to FIG. 12, 
followed by the agent performing a CPU load check 1115 to 
decide if it can accept anew client connection. If the agent 
cannot accept more work, then it notifies the main server 
1120 and if that fails, then it logs a failure and retires 1125. 
If a load check indicates sufficient resources are available, 
then the agent waits for a connection 1130. An agent 
connection 1140 with the server spawns a server thread 1145 
on the agent. A failure to create the server thread will prompt 
a security violation log entry and sub-process termination 
1165. A successful server thread creation initiates a multi- 
party key agreement, MKA, with the agent. If this is 
successful, and then a client authentication sub-process 814 
begins, see FIG. 9. Success will cause the server to 1155 
redirect the client to agent and further to 1160 spawn a client 
thread on the agent. Any failures in the previous four steps 
will prompt a security violation logging, sever notification 
and sub-process termination. 
Process for Passing Connections 

Invocation of this sub-process will prompt the agent to 
check its CPU load and decide if it should pass existing 
client 1210 off. An afiSrmative will lead the agent 1215 to 
notify the main server. Failures are logged followed by a 
retry 1220. Success allows the server and agent to exchange 
connection passing information 1225 followed by the 1235 
server sending the client redirection information. A further 
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success prompts the client to attempt to redirect 1245. If the cipher results in 16 subsequent XOR operations ind at 

client redirection is successful, then the agent thread is freed. least 16 state changes of that cleartext byte. 

Any failures in the previous three steps will initiate security states(A), a finite set of states 

violation logs and sub-process termination 1230. The third component defines the set of initial state/s of the 
Parallel Processing and Reconstitution 5 automaton. The start states are the nonempty elements of 

One of the novel features of this architecture is that client states(A) set that exist when the system is initialized 

partitions of encrypted bandwidth are assigned to individual start(A), a nonempty subset of slates(A) known as the start 

agents or multiplexed to process and decrypt. This is a new initial states 

and non-obvious appUcation of distributed algorithm tech- '"^^ f^^^*^ component defines the set of changes in state as 

nology as applied to encryption and decryption and greatly ^° ^y^^^™ becomes active. 

enhances scalabiUty. This process implementation is based trans(A), a state transition relaUon, trans(A) C states(A) x 

on automata mathematics, as is well understood. The for- ^ ac^(sig(A)) x states(A) ^ . , 

malized mathematics can be found in Nancy A. Lynches, ^^^^ component defines the set of equivalent actions 

"Distributed Algorithms", Morgan Kaufmann Publishers, ^^^^ ^^^^ ^ automata. These are commonly called 

1996. The implementation of the theory to this application is threads. 

described directly below. ^ *^sk partition 

The model is an asynchronous concurrent system. An ^ ,v. , - ^.v 

J ^fry ^ , , * • J £ J 1 * f i * (s, jt, s) IS an element of transfA). n represents an action 

Input/Output automaton IS defined as a simple type of state . • . . r.- . 

u- • u .u * • * J '.u J the causes a change in state. Smce an agent requires an input 

machine m which the transitions are associated with named „ i * * u * * *u * * * ■ ^ a 
. , J , , , . , 1 T//^ 20 packet to change state, we say that state s is considered 

actions. Actions are classified as input output or internal. I/O ^ • . -ru- \u * ♦ ^ n # 

. . , . ^ X' quiescent. This means that an agent can and will accept 

automata are combmed using a repetiUve composition i- , , , -p., u . j n f*u 

^. , f XT 1 1 * : u* u 4 additional input packets even if It has not processed all of the 

operation to form an N level automaton, which represents . V * t* n ■ i * * ui c 

./^ . . ™ . .r .I. ' * *• • previous packets. It wUl simply use a vector table function, 

the concurrent system. The input for this system operating in .-u i^ i j 

r J J- • ^ J TTi 1 * • . 1 which Will grow and shnnk as needed or a fixed sized array 

the forward direction is an encrypted IP packet. The mternal , , . , , j /i ■* j * 

action is the decryption of the packet The output is the '° ^'"f packets until they are processed (htnited to he 

* J 1 : Tn- • . It f J amount of available memory on the processor to create the 

unencrypted packet. The internal function of decryption , ^ ^ ^ . / * i* 

f' . . • M 1 1 . *u * 1 * vector or array). Unexpected packet input (corrupt packets, 

occurs on each agent IS visible only to that particular agent. . v * j i *\ iV • -i \ 

™ . r 1 rr- * . Tr malicious packcts or unsecured packets) result in silent 

There is another tuple ofTinput, process, agent response,] for j * r.u i * j •* • i *• « *e 

, ... . A 1 ' r jj 1- * ir. * destruction of the packets and a security violation flag set for 

controlling the agent. An example is [add new client IP to ,„ , u •* n * 
* , : jj m ^ * 30 the server to see when it polls the agent, 

table (server message to agent), agent adds IP entry, agent ^^^^ ^^^^ (or threads) ^at occur on the agent, 

responds with chent added message]. The input for this a * • ■ • \ i a e *u Tn. 

. , , • . 1 *• • The first is receiving control commands from the server. The 

system is the server control message, the internal action is a - a a • - , *u 

^/ . r,, . 1 J • • L J .i_ second IS sending and receivmg messages to/n*om the server, 

the processing (the agent makes a decision based on the c i . i • ^ ■ ■ j j 

, , \ r . . * • *u The final task IS participating in the encryption and decryp- 

control message) of that message. The output is the ^ i * f- u * i • * *u a 
A • J 1 f • 1 . r 35 tion of packets. Each task runs in a separate thread, 

response. An in-depth explanation of the implementation of r™_ t/^^ . . • u l i tt 

^ . 1 -.1. • A- ^ 1^ . A . 1 • The process I/O automaton is shown below. Here we 

the processmg algorithm in a distributed network using ^ *^ , , • i • r j j- 

. * f 11 J- 4i u 1 focus on the packet processing tuple in the forward direc- 

automata follows directly below. „ • i- .u- . v . u i j 

T^x^ ^ • fioD- Foi" simplicity, this assumes that the client has already 

Process automaton Pi, see FIG. 3, represents an agent in ^een authenticated and the agent is ready to begin process- 

the system. Configuration and synchronization of process Pi packets 

occurs via send and receive messages send(m)ij and receive y ^ ^^^^ ^^^^^ (v is a ciphertext packet). 

("^)J'^* Signature(Actions) 

Listed below are the definitions of the five components of Input: 

the automaton Each automata (agent) contains these five j^j^^) Initialization occurs at the moment the 

components which fully describe the automata. Since the 45 packet arrives at the agent. TTje system initializes with 

automata are identical in their external and internal charac- ^ ^^j^^ (before a packet arrives) but this case is 

terisUcs and behavior it is possible to compo«5 them mto a ^^^^^ ■^ information about the 

automaton that, once described m terms of the following five (cms characteristics or behavior, 

components, completely describes each automata within the ■ , -, 

(gjj, receive(v),.^-,veV, ISjSn, J The agent puts the received 

... ■ r . packet from the server (i) onto the internal stack to 
The first component IS the signature. The signature S of the ^^^i^ xhe restrictions that are placed on 
system consists of the set of three disjomt actions, mput ^^^^ ^^^^ ^^ j^^, ^^^^^ ^^3, ^^j^j 
process, output; in(S) represents the mput set, proc(S) ^^^^^ , j^jf 
represents the process set and out(S) represents the output internal- 
set. External actions consist of the union of in(S) U out(S) 55 , . , , , , , ^ , . , 
and local actions consist of the union of out(S) U proc(S). deade(v),, veV; The agent must decide what to do with 
Hie union of the in(S) U proc(S) U out(S) represents all P^"^^'' "° ''^'^'^P' " °' 
actions of the signature S. The external signature (also ^^T^ ■ 

known as the external interface) of S, extsig(S) is defined send(v).^, veV; veV, 1 ^kin, k;-i The agent sends the 

to be the signature (in(S),out(S),0). 60 Packet to one or more (in the case of multicast groups) 

sig(A), a signature fin^^l destinations (one of which may be itself). The 

The second component defines the set of possible states of restrictions are that there must be at least one receiver 

sig(A). Each action may or may not result in a change of ^nd it cannot be the sender, 

state for the automaton. Changes may be simple such as discard(vX-, veV; The agent discards the packet 
loading a byte onto a stack for processing or more 65 States and Start States: 

complex such as a DES-CBC computation where the packetval, a vector that is indexed by {1, . . . , n) which 

passing of a cleartext byte(octet to be more precise) to the represents the current states of the packets in V. All of the 
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packets are initialized in V as properly encrypted or currupt 
packets. Next, one by one, each packet exists in V as 
decrypted or null based on the processing decision of the 
agent. Here is an example showing the various states of two 
bytes. The first byte is a valid encrypted byte. The second s 
represents a corrupt byte. 



V(imtal) 


V(input) 


V(mteraal) 


V(output) 


10 


Null 


11110011 


11101111 


11101111 




Null 


11110000 


00000011 


null 





30 



35 



Transitions 

The transitions for the above states are as follows: 
For the vaUd byte: 

init(vX> veV-*'receive(v)y V€V-*val(j):=v-»decide(v),-, 
veV (val(j) cannot be null)-*v'-f(val(j))-*send(v').^ 
For the invalid byte: 

20 

init(v)., veV-*receive(v)jy, veV-*valQ:=v-»decide(v),., 
veV (valQ) cannot be null)-»v'=f(val(j))->val(j):« 
null-*discard(v'),- 
Tasks: 

The two tasks that are operating as separate threads are 
send(v)^.^ and discard(v),- 

There are no further state changes to val(j) when these 
tasks are executed (until the next round). 
Sample successful execution on an automata (Agent) 
[ Jrepresents the packet buffer on the agent 
ffddeeaabbcc represents the ciphertext block (also works for 

stream input) 
hello world! represents the plaintext block 
X denotes the empty sequence 

[X], rcceive(ffddeeaabbcc),-y,[ffdde6aabbcc],decide 

(ffddeeaabbcc), J^hello world!], send[hello world! ]^ 
[X] . . . 

Sample unsuccessful execution on an automata (Agent) 
[X], receive(ffddeeaabbcc),-y,[ffddeeaabbcc], decide 

(ffddeeaabbccX-^-^null], discard[nullX^ [X] . . . . 

The synchronization process is a necessary part of the 
automaton or complete system. The agent is a process 
running on a distributed system. Each client is a process 
running in the system. The gateway server is in the middle. 
FIG. 6 shows these host topology relationships. In order to 
make the single -to-many connection (1 client, N agents) 45 
which is the implementation of the bandwidth "slicing" 
technique, portions of the bandwidth arc equally divided 
among the N agents for processing. 

Since the server is responsible for authenticating and 
connecting the parties, it serves as the synchronizer. The so 
agent and chent processes have to be synchronized or else 
they cannot communicate. Encrypting and decrypting can be 
accomplished without this but it creates a bottleneck at the 
gateway. This is why current secure network solutions are 
using hardware to help mitigate that bottleneck. None of 
those existing hardware solutions deal with high volume 
traffic by any other means but by adding more hardware into 
the system. Thus, this software solution overcomes the need 
for addition encryption/decryption hardware. 

The system architecture maintains certain conditions or 
restrictions, which must exist to enable optimal composition 
of automata. The true value in this system is seen when 
automata are composed. The fact that each automaton exhib- 
its identical characteristics and behavior makes it possible 
for their I/O executions to be composed thereby increasing 
the number of operations that can be performed in the same 65 
time slice. This conclusion is based on Cartesian multipli- 
cation and implemented in this architecture. According to 



Lynch, reference cited above, there are three restrictions that 
must be met in order for this type of composition to be valid. 
These necessary conditions are designed to exist in the 
system for the wire -speed performance levels. 

First, there must be a total independence among the 
agents. If one or more agents rely on each other for data 
transfer and one of them went offline, then the other(s) 
would stop working. In other words, with respect to the 
internal actions of any two automata in the system, one 
cannot affect the other in any way. A violation of this 
condition would introduce chaos into the system via the 
avalanche effect if agent after agent began dropping out of 
the system on the account of one failure. This is shown 
pictorially in 1320 of FIG. 13. 

Second, the output of one agent cannot become the input 
of one or more other agents because to do so would 
introduce feedback into the system which will bring the 
entire system (and probably the network) to a hall in a short 
period of time. It will be possible for one agent to forward 
data to another after it's been decrypted since in that case the 
receiving agent is treated as an end host instead of an agent. 
This is pictorially depicted at 1320. 

Thirdly, it is required that a finite number of agents exist 
in the system. Since infinity is a large number, the practical 
limitation of this condition is small and at least one agent can 
be generated to satisfy the zero condition. This is pictorially 
depicted at 1330. FIG, 14 shows the overall relationship 
between automata and the automaton. 

The foregoing advantages of the present invention were 
proved in laboratory performance tests, which show the 
effectiveness of the present invention as compared to con- 
ventional architectures. The test configurations are shown in 
FIG. 1 and FIG. 2, FIG. 1 shows the Test Case 1, where a 
configuration employed a gateway server, which decrypted 
trafi&c before forwarding to the final destinations. In this 
configuration clientl 110 and chent2 120 opened secure 
communication with destination clientl 125 and destination 
client2 130. Decryption was accomplished on the Main 
Server 120 and then decrypted packets were sent to final 
destinations. The average CPU usage, system load and 
packets per second are shown below for over 100 test runs. 
DET Test Case 1 
Server Running on Solaris 
1 Agent Running on Main Server 
3 NT Clients/1 98 Client 
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FIG. 2 shows The Test Case 2 configuration employed the 
invention architecture. This configuration performs the net- 
work address translation at the main server 220 and hands 
the decryption work off to the agents running on the final 
destination hosts 225 and 230. The results given below show 
a roughly double packet per second throughput perfor- 
mance. The average CPU usage, system load and packets per 
second are shown below for more than 100 test runs. 
DET Test Case 2 
Server Running on Solaris 

2 Agents Running on NT 

3 NT Clients/1 98 Client 
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While the iavention has been described in connection with 
what are presently considered to be the most practical and 
preferred embodiments, it is to be understood that the 
invention is not limited to the disclosed embodiments, but 
rather is intended to cover various modifications and equiva- 
lent arrangements which are included with the spirit and 
scope of the following claims. 
I claim: 

1. A software architecture for conducting a plurality of 
cryptographic sessions over a distributed computing 
environment, comprising: 

a registration entity or registry residing within a main 
server entity; 

an agent server entity communicating with said main 
server; 

a client entity communicating with said main server and 
agent server; 

a plurality of distributed networked computers providing 

a mechanism for executing said main server entity, 

agent server entity, and client entity; 
a defined protocol for initiating secure communication 

between the main server and agent server; over said 

network; and 

a system for providing one or more communication 
sessions among the main server, agent server and client 
entity for implementing a client decrypted bandwidth 
reconstitution which enables the recombination of indi- 
vidual parts of the decrypted client bandwidth among N 
agents processing in parallel. 

2. A system according to claim 1 comprising: 
means for discovering said agent servers; 

means for determining an available processing bandwidth 

of the main server and agent servers 
means for registering said main server and available agent 

server with said registration entity. 

3. A system according to claim 1 wherein said system for 
providing one or more communication sessions among the 
main server, agent server and client entity comprises parti- 
tioning the client bandwidth among N agents such that every 
agent receives 1/N of the encrypted bandwidth. 

4. A system according to claim 1, wherein the system for 
establishing communication sessions among the main server 
agent and client comprises automata composed according to 
the restrictions of: 

limiting control communication between the agent and 
server only; 

accepting input/output from a gateway server only; and 
requiring a finite number of automata to exist. 

5. An end to end encryption/decryption system imple- 
mentable in software for a distributed network comprising: 

one or more networked computers; 

a main server communicating with said networked 

computers, said main server including a registration 

entity; 

one or more agent servers communicating with said main 
server for defining one or more session keys for estab- 
lishing secure connections with said networked com- 
puters such that when the main server receives notifi- 55 
cation that an agent server is saturated, the main server 
finds an alternate agent server and passes the session 
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key to the alternate agent £er\'er and notifies 2 corre- 
sponding networked computer to redirect the secure 
session to the alternate agent server. 

6. A method for implementing a scalcable software crypto 
system between a main server and one or more agent servers 
communicating with one or more clients such that perfor- 
mance of the crypto system is increased to meet any demand 
comprising providing a secure communication between the 
main server, agent server, and one or more clients such that 
communication between the main server and agent server 
automatically enlists additional agent servers to support 
incremental secure sessions so as to maintain performance at 
a desired level. 

7. A method for conducting a plurality of cryptographic 
sessions in parallel over a distributed computing environ- 
ment including one or more agents, a server and one or more 
cUents comprising: 

establishing a secure cryptographic session environment; 

initiating secure communication with a main server and 
define one or more session keys with which to establish 
secure sessions; and 

transferring secure sessions, such that the main server and 
one or more agent servers become enabled to receive 
secure sessions from clients; 

such that establishing, initiating, and transferring secure 
cryptographic sessions provide N simultaneous scal- 
able secure cryptographic sessions among agents, 
server and clients. 

8. The method of claim 7 wherein establishing a secure 
cryptographic session comprises: 

registering one or more agent servers with the main 
server; defining one or more session keys, 

such that the main server and agent servers become 
enabled to receive secure connections with the chents. 

9. The method of claim 7 wherein initiating secure 
communication comprises; 

connecting one or more clients to the Main Server for 

authenticating; and 
determining if the Main Server can accept a new session 

based on current available processor bandwidth of said 

main server; and 
agreeing on a secret session key with the one or more 

clients; and 

enabling one or more available Agent Servers to become 
unblocked and participate in a multiparty key exchange 
between a Client, Main Server and Agent Server when 
the Main Server has insufBcient resources to service the 
session; and 

denying a client connection when main server and agent 
servers are unavailable; and 

maintaining a list of connections or sessions and associ- 
ated session information and session keys; and 

terminating encrypted session communication upon suc- 
cessfully transferring a session from main server to one 
or more agents. 

10. A method for distributed encryption/decryption imple- 
mented in software across a computer network employing a 
distributed automaton comprising M automata for servicing 
a plurality of N simultaneous crypto sessions which provides 
bandwidth scalability Umited only by the M automata com- 
prising: 

sharing spare CPU cycles of the computer network for 
encrypting and decrypting communication to provide N 
simultaneous secure session among said network of 
computers. 
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